Add pre-release E2E CI: full-path release gate with layered per-leg judging - #1303

The bootstrap script only reads these as plain text for `claude --print`, so the
extension was never load-bearing. Markdown fits better -- they are natural-language
instructions, and .md renders headings/lists/code blocks and matches the repo's
SKILL.md style. Content is semantically unchanged (same flags, overrides, and
hard-constraint wording -- this is an automated release gate); only the formatting
was reorganized into sections. Updates the two bootstrap references and the build
job's cp glob from *.txt to *.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Confirmed against the Primus-SaFE codebase that the create-workload body
embeds WorkloadSpec inline, whose top-level integer-seconds `timeout` field
is enforced by WorkloadTTLController for all workload kinds (incl. Authoring),
counted from dispatch time. Replaces the placeholder `activeDeadlineSeconds`
default (a k8s-Job convention that is not the Authoring create-API field).

DEADLINE_FIELD stays overridable; "" falls back to workspace maxRuntime /
poll-side GLOBAL_TIMEOUT_S.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…inD)

Verified on a real SaFE privileged Authoring pod (MI355X x8, rocm/pytorch
base image) that the base image ships no docker/dockerd, and fixed the
nested-container path against what actually works:

bootstrap-pre-release.sh
  * add ensure_dockerd(): apt-get install docker.io, then start a pod-local
    dockerd detached via setsid with --storage-driver=vfs (no systemd in the
    pod; overlay-on-overlay fails). Called before fanning out docker legs.

docker-run-hyperloom.sh
  * GPU index -> renderD is stride 8, not +1: RD=128+GPU_INDEX*8 (verified
    GPU i == renderD(128+8i) via /sys/class/drm + rocm-smi --showbus).
  * --group-add video FAILS (pod /etc/group has no video/render names); use
    numeric device-node GIDs from `stat -c %g` instead.
  * isolate a single card via /dev/kfd + one renderD node only (drop the
    cardN device; card numbering is not guaranteed aligned to GPU order),
    plus --security-opt seccomp=unconfined and HIP_VISIBLE_DEVICES=0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Policy change: after judging (and on job cancel), STOP each SaFE workload
instead of DELETE. Stop frees the GPUs immediately -- so a 3h leg that
finished early does not idle-hold its card until the `timeout` deadline --
while keeping the workload record and pod filesystem for post-hoc inspection.
SaFE exposes no start/restart, so Stopped records are cleaned up manually.

Verified 2026-08-27: POST /api/v1/workloads/{id}/stop exists and returns 200.

  * poll: add stop_workloads() at the end, dedup by workloadId (the 4 docker
    legs share one host workload).
  * workflow: cancel step switched from DELETE to POST .../stop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Rename every ${{ secrets.* }} / ${{ vars.* }} reference to carry a PRE_E2E_
prefix so this release-gate CI's config never collides with the existing
per-PR CI (ci-e2e.yml) secrets/variables in the same repo. GITHUB_TOKEN is
left as-is (built-in, cannot be renamed).

Only the reference names change; the in-workflow env: keys handed to the
dispatch/poll scripts (SAFE_API_BASE, NFS_ROOT, ...) are unchanged, so no
script edits are needed.

Owner action: create the repo Secrets/Variables under the new PRE_E2E_ names.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Run the release gate on the OPEN PR (before merge), not after merge. A
push-to-main trigger is deliberately removed so a bad release is blocked
while the PR is open and merging does not re-run it.

resolve.decide now classifies against the PR base branch instead of HEAD~1:
version bump vs base -> FULL (8 legs); CI-logic-only change -> SCRIPTS-ONLY
(4 fast 3h legs). Uses BASE_SHA/BASE_REF from the event; checkout fetch-depth
raised to 0 so the base commit is present for the diff/version compare.

poll: PR number now comes straight from the event (PR_NUMBER) with the
commit->PR reverse lookup kept as a workflow_dispatch fallback; per-leg
commit statuses target the PR head sha (github.sha is the merge commit).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Move the release-gate jobs off the shared Hyperloom-e2e-ci runner (which the
per-PR smoke test ci-e2e.yml also uses, causing build to queue behind it) onto
their own SaFE-hosted AutoscalingRunnerSet `hyperloom-pre-e2e-ci`. All three
jobs (resolve/build/run) now target it, so the pre-release gate no longer
contends with the per-PR CI for runner slots.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…-hfjjz

The bare install name hyperloom-pre-e2e-ci did not pick up jobs (queued >2min);
the SaFE AutoscalingRunnerSet's listener pod is hyperloom-pre-e2e-ci-hfjjz-*,
so the runner scale set name carries the -hfjjz suffix. Point runs-on at it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Switch runs-on from the SaFE ARC scale-set (hyperloom-pre-e2e-ci-hfjjz) to
a standard self-hosted runner on crsuse2-m2m-061 (label
hyperloom-pre-e2e-baremetal). The SaFE runner-proxy image predates the
USER_APIKEY auth support (PR #662, 2026-07-10), so it never sends the
injected platform key and every runner start 401s. A direct GitHub runner
bypasses runner-proxy entirely.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two bare-metal fixes:
- dispatch: report create_workload failures to stderr, not stdout. The call
  runs inside wid=$(create_workload ...) command substitution, so a stdout
  message was captured into $wid and never reached the CI log -- the job
  failed with exit 1 and no visible reason. stderr surfaces the real HTTP
  status + SaFE API body.
- build: add an idempotent step to mkdir+chown the CI NFS root. The runner
  runs as 'ubuntu', not in the group owning /shared_nfs, so it cannot create
  the root. Self-heals across runner reinstall / NFS remount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SaFE derives the k8s object name from displayName. CI_VERSION contains '+'
(1.0.0.dev...+ci), which is not a legal RFC 1123 subdomain char, so every
create_workload 422'd ('metadata.name Invalid value'). Fold illegal chars to
'-', lowercase, collapse/trim dashes before POSTing. The poll step keys off
the DISPATCH_MAP (leg -> workloadId), not the name, so this is safe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SaFE's vworkload admission webhook enforces a stricter rule than RFC 1123:
1-44 chars, must start with an alphabetic char, end alphanumeric, lowercase
[a-z0-9-]. The old 'e2e-<CI_VERSION>-<leg>' name embedded the full version
(1.0.0.dev...+ci) -> 47 chars AND a leading-digit/'.'/'+' after the prefix.
Build the name as 'e2e-<leg>-<6-hex CI_VERSION hash>' so the leg stays intact
and never collides across runs, and harden the sanitizer (strip leading
non-alpha, cap 44, re-trim trailing dash).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nd, resources)

Six fixes surfaced by the first live run (CI_VERSION 1.0.0.dev202608271014+ci):

1. 401 subscription-key: AMD's APIM gateway (llm-api.amd.com) rejects the bearer
   key alone -- it needs an Ocp-Apim-Subscription-Key header. Thread a new
   ANTHROPIC_CUSTOM_HEADERS through workflow -> dispatch env -> bootstrap .env; the
   CLI expands ${ANTHROPIC_API_KEY} so one secret covers bearer + subscription.

2. kind Authoring -> PyTorchJob: the Authoring mutating webhook overwrites
   EntryPoints to `sleep infinity`, so our bootstrap never auto-ran. PyTorchJob is
   not in that mutate switch and honors the submitted entrypoint; privileged/8-GPU/
   useWorkspaceStorage/timeout are all kind-agnostic. E2E_DOCKER_HOST now travels in
   the workload env (a `VAR=1;` command prefix wouldn't export into the entrypoint).

3. root permissions: `claude --print` is fail-closed and dies with no approver ->
   add --dangerously-skip-permissions; it refuses under root unless IS_SANDBOX=1
   (SWSPLAT-42390), matching Hyperloom's own kernel-agent. Also cd into the leg root
   so claude finds the workspace .env instead of blocking at '/'.

4. minimal base image: install Node/npm (claude CLI) and jq up front -- the SaFE
   rocm/pytorch image ships neither.

5. host resources: raise the privileged docker host to mem 2048Gi / ephemeral
   1792Gi (ref sglang-kimik3-2). The first run was EVICTED at ephemeral 200Gi:
   vfs storage x 8 ROCm images has no layer dedup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…repo var needed)

The Ocp-Apim-Subscription-Key header value is just the API key (${ANTHROPIC_API_KEY},
expanded in-pod by Hyperloom's parse_custom_headers), carrying no new secret -- the
key already reaches the pod via ANTHROPIC_API_KEY_B64. So instead of requiring a
PRE_E2E_ANTHROPIC_CUSTOM_HEADERS repo variable (a "forgot to set it -> 401" trap),
bootstrap now defaults ANTHROPIC_CUSTOM_HEADERS to
`Ocp-Apim-Subscription-Key: ${ANTHROPIC_API_KEY}` whenever a gateway base URL is set,
matching Hyperloom's .env.template and hyperloom-setup SKILL.md exactly. An externally
supplied header still overrides; direct api.anthropic.com emits no header.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The release-gate legs hold up to 8 GPUs for as long as 14h and block the release,
so they must not be starved behind dev workloads. SaFE's scheduler orders the queue
by Spec.Priority (int; High=2/Med=1/Low=0 per Primus-SaFE constant.go; the webhook
clamps to [0,2]). Add `priority: $prio` to the create-workload body, default 2,
overridable via PRIORITY.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… rerun)

Make "push a fix -> the stale run is torn down and the fresh code reruns from scratch"
automatic:

* concurrency.cancel-in-progress: true, group keyed per-PR. GitHub now cancels the
  in-flight run on a newer commit to the same PR; the run's `if: cancelled()` step
  stops its SaFE workloads (freeing GPUs) before the new run dispatches. Previously
  (cancel-in-progress: false) a new push queued behind a run built from superseded
  code, and stale legs kept their GPUs until they timed out.

* dispatch writes DISPATCH_MAP INCREMENTALLY (record_dispatch appends after every
  successful create, seeded with an empty map up front) instead of once at the end.
  With cancel-in-progress a cancel can land mid-dispatch; the end-only write would
  leave already-created workloads absent from the map and leak their GPUs. Now the
  cleanup step always sees every workload created so far.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ded)

claude --print is a single non-interactive turn: the demo skill backgrounds
`optimize` (setsid nohup) and returns immediately, so run.sh exited 0 and SaFE
marked a false "Succeeded" after ~6min for a 3h/12h benchmark that never wrote
reports/final.json.

run_leg now blocks after the demo turn until the backgrounded optimize reaches a
terminal state: it globs the newest nested session dir under $session that holds a
state.json (make_session_dir creates $session/<model>/<UTC_ts>-<rand8>/ and re-pins
INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR only inside the CLI process, so the glob is
authoritative), re-points the poll's .session_dir pin at it, and polls until
reports/final.json exists (or state.json has a terminal stop_reason + a short grace).
Returns 0 iff final.json exists (PASS/FAIL by gain stays the poll's job); returns 1
on startup grace (optimize never launched), hard deadline (hours*3600+3600), or a
terminal stop_reason with no report -> SaFE marks Failed, not a false Succeeded.

Fixing the .session_dir pin also fixes a second latent bug: the poll looked for
$session/reports/final.json (parent), but the report lives in the nested run dir.

dispatch: widen SaFE pod timeout (DEADLINE_3H_S 14400->16200, DEADLINE_12H_S
46800->48600) so the pod hard-timeout stays strictly greater than bootstrap's own
in-pod wait deadline; otherwise SaFE could pre-empt the pod mid-wait and lose the
clean return-1 path. Ordering per leg: bootstrap deadline < SaFE pod timeout < poll
GLOBAL_TIMEOUT_S (50400s).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eak)

The concurrency.cancel-in-progress knob only cancels the GitHub JOB; it does NOT
reliably stop the SaFE PyTorchJob pods a superseded/failed run already created. The
`if: cancelled()` cleanup only gets a short grace window, and a job that FAILS (not
cancels) after dispatch skipped cleanup entirely. Verified: 3 `Running` e2e workloads
(incl. an 8-GPU docker host) leaked from a dead run and idle-held their cards, starving
the next run.

Correct order is the reverse of "cancel job -> hope the pod stops": a NEW run now
STOPS every stale e2e-* workload (this workspace, non-terminal phase, not this run's own
VERSION_TAG) up front via GET /workloads + POST /stop, frees the GPUs, then dispatches.
The old run's poll then sees phase=Stopped and judges those legs FAIL -- the GitHub job
ends as a CONSEQUENCE of stopping the pod, not the other way round. reap is resilient to
an unreachable API (skips, never aborts under set -e).

Also broaden the in-run cleanup step from `if: cancelled()` to
`if: cancelled() || failure()` so a job that fails after dispatch stops its own
workloads too (idempotent; poll's stop_workloads handles the success path; the next
run's reap is the final backstop).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ase-e2e run

Temporary, for testing the pre-release E2E gate end-to-end on all 8 legs. The PR's
resolve job classifies a version bump vs main as a FULL run (all 8 legs) instead of
the scripts-only 4-leg scope. 1.0.1a0 is a legal PEP 440 alpha so the CI_VERSION
(1.0.1a0.dev<ts>+ci) and wheel build stay valid. REVERT to 1.0.0 before merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…vior

Reformat test_pre_release_gate_orchestration and test_pre_release_stall_liveness
so ruff format --check passes in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

VERSION_TAG now hashes CI_VERSION with GITHUB_RUN_ID so dispatch reap can
stop pods from a superseded push even when the wheel version is unchanged.
Poll detects a newer queued workflow run for the same PR branch, releases
the runner without stopping workloads, and leaves teardown to the successor.

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Align poll gate with optimize CLI exit 0: final.json must carry a terminal
stop_reason in the success set (time_exhausted, target_reached, etc.).
TARGET_GAIN still flows to the demo skill at 100 but no longer gates PASS.

Co-authored-by: Cursor <cursoragent@cursor.com>

Judge PASS/FAIL from state.json stop_reason instead of final.json, and
complete bootstrap when optimize exits cleanly even if final.json is late.
Sync 3h to framework-only (no --no-framework-agent, 0.90 framework pct),
align 12h prompts with the 14b skill, and raise demo re-drive budget to 5.

Co-authored-by: Cursor <cursoragent@cursor.com>

Normalize quote style so ruff format --check passes in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>